Genetic differentiation after founder events: an evaluation of FST estimators with empirical and simulated data

Question: Given a model of speciation by host shift, how do different marker types and measures of genetic differentiation compare in detecting reproductive isolation between a small, recently founded population and its large source population? Data incorporated: We used empirical data from brood parasitic indigobirds (Vidua spp.) as well as simulated mitochondrial and microsatellite data for a founding event with immediate cessation of gene flow and subsequent population growth. Method of analysis: We evaluated the performance of different estimators (θST, RST, and ΦST) in detecting population differentiation and compared our simulation results with previously collected empirical data. Conclusions: With much greater variance, RST was less reliable than θST in detecting incipient differentiation in microsatellite data. Negative RST values for individual loci in both the empirical data and up to 20% of simulation replicates occurred when genetic drift changed allele frequencies but not allele size variance. Both ΦST and θST reliably detected genetic isolation from simulated sequence data. As bottlenecked populations regained variability, however, θST values decreased due to a negative correlation with polymorphism, an effect also observed in the empirical data. Given this effect, the standardization of differentiation estimates relative to population heterozygosity is recommended for both types of markers, although this approach does not correct for the reduction of θST due to allele size homoplasy in microsatellites. In contrast, ΦST values gradually increased over time as novel mutations increased the divergence between haplotypes. Our analyses suggest that for microsatellite data, θST outperforms RST in detecting recently established reproductive isolation, whereas ΦST is preferable for sequence data regardless of divergence time.